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Abstract 

Many computer vision and image processing problems can be posed as solving 
partial differential equations (PDEs). However, designing PDE system usually re- 
quires high mathematical skills and good insight into the problems. In this paper, 
we consider designing PDEs for various problems arising in computer vision and 
image processing in a lazy manner: learning PDEs from real data via data-based 
optimal control. We first propose a general intelligent PDE system which holds 
the basic translational and rotational invariance rule for most vision problems. By 
introducing a PDE-constrained optimal control framework, it is possible to use the 
training data resulting from multiple ways (ground truth, results from other meth- 
ods, and manual results from humans) to learn PDEs for different computer vision 
tasks. The proposed optimal control based training framework aims at learning a 
PDE-based regressor to approximate the unknown (and usually nonlinear) map- 
ping of different vision tasks. The experimental results show that the learnt PDEs 
can solve different vision problems reasonably well. In particular, we can obtain 
PDEs not only for problems that traditional PDEs work well but also for prob- 
lems that PDE-based methods have never been tried before, due to the difficulty 
in describing those problems in a mathematical way. 
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1. Introduction 

The wide applications of partial differential equations (PDEs) in computer vi- 
sion and image processing can be attributable to two main factors [HI. First, PDEs 
in classical mathematical physics are powerful tools to describe, model, and sim- 
ulate many dynamics such as heat flow, diffusion, and wave propagation. Second, 
many variational problems or their regularized counterparts can often be effec- 
tively solved from their Euler-Lagrange equations. Therefore, in general there are 
two types of methods for designing PDEs for vision tasks. For the first kind of 
methods, PDEs are written down directly (e.g., anisotropic diffusion [[21, shock 
filter [3], based on some understandings on the properties of mathematical oper- 
ators and the physical natures of the problems, and curve-evolution-based equa- 
tions flU). The second kind of methods basically define an energy functional and 
then derive the evolution equations by computing the Euler-Lagrange equation of 
the energy functional (e.g., total-variation-based variational methods [IIljSlBTll). 
In either way, people have to heavily rely on their intuition on the vision tasks. 
Therefore, traditional PDE-based methods require good mathematical skills when 
choosing appropriate PDF forms and predicting the final effect of composing re- 
lated operators such that the obtained PDEs roughly meet the goals. If people do 
not have enough intuition on a vision task, they may have difficulty in acquiring 
effective PDEs. For example, although there has been much work on PDE-based 
image segmentation ll8l[l9l [fT0l [ [ TT | . the basic philosophy is always to follow the 
strong edges in the image and also require the edge contour to be smooth. Can we 
have a PDF system for objective detection (Fig. [TJ that locates the object region 
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Figure 1: Can we design a PDE system which can detect the object of interest (e.g., the helicopter 
in the left image) and does not respond if the object is absent (e.g., the right image)? 

if the object is present and does not respond if the object is absent? We believe 
that this is a big challenge to human intuition and is much more difficult than 
traditional segmentation tasks if a PDE-based method is required, because it is 
hard to describe an object class, which may have significant variation in shape, 
texture and pose. Without using additional information to judge the content, the 
existing PDEs for segmentation, e.g., [ITOl . always output an "object region" for 
any non-constant image. In short, current PDE design methods greatly limit the 
applications of PDEs to a wider and more complex scope. This motivates us to 
explore whether we can acquire PDEs that are less artificial yet more powerful. 
In this paper, we give an affirmative answer to this question. We demonstrate that 
learning particular coefficients of a general intelligent PDE system from a given 
training data set might be a possible way of designing PDEs for computer vision 
in a lazy manner. Furthermore, borrowing this learning strategy from machine 
learning can generalize PDEs techniques for more complex vision problems. 

Inspired by the electromagnetic field theory and Maxwell's equations [fT2l . 
we assume that the visual processing has two coupled evolutions in different scale 
spaces: one is in the image scale space, which controls the evolution of the output, 
and the other is in the indicator scale space that helps collect the global informa- 
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tion to guide the evolution in the image scale space. In this way, our general 
intelligent PDE system consists of two coupled evolutionary PDEs. Both PDEs 
are coupled equations between the image and indicator, up to their second order 
partial derivatives. Another key idea of our general intelligent PDE system is to 
assume that the PDEs that are sought could be written as combinations of "atoms" 
which satisfy the general properties of vision tasks. As a preliminary investiga- 
tion, we utilize all the translational and rotational invariants as such"atoms" and 
propose the general intelligent PDE system as a linear combination of all these 
invariants lfT3l . Then the problem boils down to determining the combination 
coefficients among such "atoms". 

The theory of optimal control lfT4l has been well developed for over fifty years. 
With the enormous advances in computing power, optimal control is now widely 
used in multi-disciplinary applications such as biological systems, communication 
networks and socio-economic systems etc ifTSl . Optimal design and parameter es- 
timation of systems governed by PDEs give rise to a class of problems known 
as PDE-constrained optimal control [16]. In this paper, a PDE-constrained opti- 
mal control technique as the training tool is introduced for our PDE system. We 
further propose a general framework for learning PDEs to accomplish a specific 
vision task via PDE-constrained optimal control, where the objective functional 
is to minimize the difference between the expected outputs and the actual outputs 
of the PDEs, given the input images. Such input-output image pairs are provided 
in multiple ways (e.g., ground truth, results from other methods or manually gen- 
erated results by humans) for different tasks. Therefore, we can train the general 
intelligent PDE system to solve various vision problems which traditional PDEs 
may find difficult or even impossible. 
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In summary, our contributions are as follows: 

1. Our intelligent PDE system provides a new way to design PDEs for com- 
puter vision. Based on this framework, we can design particular PDEs for 
different vision tasks using different sources of training images This may 
be very difficult for traditional PDE design methods. However, we would 
like to remind the readers that we have no intention to beat all the existing 
approaches for each task, because these approaches have been carefully and 
specially tuned for the task. 

2. We propose a general data-based optimal control framework for training the 
PDE system. Fed with pairs of input and output images, the proposed PDE- 
constrained optimal control training model can automatically learn the com- 
bination coefficients in the PDE system. Unlike previous design methods, 
our approach requires much less human wits and can solve more difficult 
problems in computer vision. 

The rest of the paper is structured as follows. We first introduce in Section |2] 
the general intelligent PDE system. In Section |3] we utilize the PDE-constrained 
optimal control technique as the training framework for our intelligent PDE sys- 
tem. Then in Section |4] we evaluate our intelligent PDE system with optimal 
control training framework by a series of computer vision and image processing 
problems. Finally, we give concluding remarks and a discussion on the future 
work in Section [51 

'Similar idea also appeared in (Tl]. But in that work, the authors only train special PDEs 
involving the curvature operator for basic image restoration tasks. In contrast, our work here 
proposes a more unified and elegant framework for more problems in computer vision. 
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2. General PDE System for Computer Vision 

2.1. Electromagnetic Field vs. Image Evolution 

Electromagnetism is the force that causes the interaction among electrically 
charged particles. The areas in which electromagnetic interaction happens are 
called the electromagnetic fields. But in physics, electrically charged objects were 
first thought to produce two types of fields associated with their charge property: 
An electric field and a magnetic field. Over time, it was realized that the electric 
and magnetic fields are better thought of as two parts of a greater whole - the 
electromagnetic field [ fT2| . It affects the behavior of charged objects in the vicin- 
ity of the field. The electromagnetic field extends infinitely throughout space and 
describes the electromagnetic interaction. The theoretical implications of electro- 
magnetism also led to the development of special relativity by Albert Einstein in 
1905. 

In this paper, inspired by this fundamental force of nature, we consider the 
image evolution in a similar way. For a target image signal u{t), different from 
most traditional ways, which only consider the evolution in the image scale space, 
we define a companion signal v{t) named the indicator signal. It changes with 
time and guides the evolution of u{t) by collecting large scale information in the 
image. In this way, these two signals evolve in two coupled scale spaces. 

2.2. The Intelligent PDE System 

Similar to Maxwell's equations [fT2ll . which are a set of PDEs describing how 
the electric and magnetic fields relate to their sources and how they develop with 
time, we propose a general PDE system for the evolution of our coupled signals. 
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The space of all PDEs is infinitely dimensional. To find the right form, we start 
with the properties that our PDE system should have, in order to narrow down the 
search space. We notice that translationally and rotationally invariant properties 
are very important for computer vision, i.e., in most vision tasks, when the input 
image is translated or rotated, the output image is also translated or rotated by 
the same amount. So we require that our PDE system is translationally and ro- 
tationally invariant. According to the differential invariant theory [[T3l . the form 
of our PDEs must be functions of the fundamental differential invariants under 
the group of translation and rotation. The fundamental differential invariants are 
invariant under translation and rotation and other invariants can be written as their 
functions. We list those up to second order in Table [1] where some notations can 
be found in Table [2j In the sequel, we shall use {inw j{u, f )}]io ^'^ refer to them 
in order. Note that those invariants are ordered with u going before v. We may 
reorder them with v going before u. In this case, the j-th invariant will be referred 
to as imj{v, u). So the simplest choice of our general PDE system is the linear 
combination of the differential invariants, leading to the following form: 








(x,y,t) G Q, 

(x,y,t) G r, 



(1) 



< 








{x,y,t) G Q, 
{x,y,t) G r, 
{x,y) G n, 



where 



F{u,v,{a,}]%) 
Fiv,u,{b,}]io) 



E]=o^i(^)inVi(f,M), 



(2) 
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f2 is the rectangular region occupied by the input image /, T is the time that the 
PDE system finishes the visual information processing and outputs the results, 
and /„ and f^, are the initial functions of u and t>, respectively. The meaning of 
other notations in ([T]) can be found in Table |2] For computational issues and the 
ease of mathematical deduction, / will be padded with zeros of several pixels 
width around it. As we can change the unit of time, it is harmless to fix T = 
1. {aj(t)}]f^Q and {hj{t)Y^Q are sets of functions defined on Q that are used to 
control the evolution of u and v, respectively. As Vm and H„ change to RVm 
and RH„R^, respectively, when the image is rotated by a matrix R, it is easy 
to check the rotational invariance of those quantities. So the PDE system ([T]) 
is rotationally invariant. Furthermore, the following proposition implies that the 
control functions aj{t) and hj{t) can be functions of t only. 

Proposition 2.1. Suppose the PDE system ([7| is trans lationally invariant, then 
the control functions {ajjjio {bj}]=o f^^^t be independent of{x, y). 



The proof of Proposition 2.1 is presented in Appendix A 



3. Training the PDE System via Data-based Optimal Control 

In this section, we propose a data-based optimal control framework to train 
the intelligent PDE system for particular vision tasks. 

3.1. The Objective Functional 

Given the forms of PDEs shown in ([T]), we have to determine the coeffi- 
cient functions aj{t) and hj{t). We may prepare training samples {(/m, Om)} 



M 

m,=l' 



where is the input image and Om is the expected output image, and compute 
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Table 1: Translationally and rotationally invariant fundamental differential invariants up to the 
second order. 



J 


imj[u, V) 


0,1,2 


1, V, u 


3,4 


\\T—r 119 9 1 9 119 9 i 9 

II v^;|r = + < II Viiir ^K + s 


5 


(Vl')'^Vlt = l'a;lia; + VyUy 


6,7 


tr(H^) = f j^a; + Vyy, tr(H„) = Ma;^ + Uyy 


8 




9 


(Vv)^HuVf = v^Uxx + 2vxVyUxy + t'^iiyj, 


10 


(Vt')^H^VU = t'xIiaJ^'xX + (t^X^ + VyUx)Vxy + VyUyVyy 


11 


(Vi;)'^H„VM = V^UxUxX + {VxUy + f + VyUyUyy 


12 


{Vu)'^HyVu = ulVxx + 2UxUyVxy + 


13 


(V-uj^HuV-u = M^Mxx + 2uxUyUxy + M^^y 


14 




15 


tr(H^Hy) — '^xx'^xx ~l~ "^^xy^xy ~l~ ^yy^yy 


16 


HK)=^l + ^<y + 4 
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Table 2: Notations. 



n 


An open bounded region in 


dn 


Boundary of Vl 




(x, y) E ^l, spatial variable 


t 


t E (0, T), temporal variable 


Q 


n X (0,T) 


r 


an X (o,r) 




The area of a region 




Transpose of matrix (or vector) 




norm 


tr(.) 


Trace of matrix 


Vm 


Gradient of u 


H„ 


Hessian of u 


P 


p = {(0,0), (0,1), (1,0), (0,2), 


(1,1), 


(2, 0)}, index set for partial differentiation 



the coefficient functions that minimize the following functional: 

= lElii/oK(^,2/,i)-0™Nfi (3) 

where Umix,y, 1) is the output image at time t = 1 computed from ([T]) when 
the input image is /„, and Aj and fii are positive weighting parameters. The first 
term requires that the final output of our PDE system be close to the ground truth. 
The second and the third terms are for regularization so that the optimal control 
problem is well posed, as there may be multiple minimizers for the first term. 
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(4) 



3.2. Solving the Optimal Control Problem 

Then we have the following optimal control problem with PDE constraints: 

argmin J{{u^YJ^^^, {aj}]to. {bj}f=o) 

f -F(M,t;,{a,}]io) = 0, (x,2/,t)GQ, 

u{x,y,t) = Q, {x,y,t)eT, 

u\t=o = fu, {x,y) e Q, 

^-Fiv,u,{bj}]%) = 0, ix,y,t)EQ, 

v{x,y,t) = 0, {x,y,t)eT, 

^ v\t=o = fv, ix,y) e n. 

By introducing the adjoint equation of (|4]), the Gateaux derivative of J can be 
computed and consequently, the (local) optimal {aj}^^Q and {bj}^^Q can be com- 
puted via gradient-based algorithms (e.g., conjugate gradient). Here, we give the 
adjoint equation and Gateaux derivative directly: 



s.t. < 



3.2.1. Adjoint Equation 



^ + E{Um, Vm, '-Pm, 4>m) 


= 0, 


{x,y,t) G Q, 


<fm = 0, 




{x,y,t) e r, 


(Pm\t=l = Om — Um{i), 




{x,y) e 


^ + E{Vm, Um, <Pm, V^m) 


= 0, 


ix,y,t) e Q, 


(pm = 0, 




ix,y,t) e r, 


(j)m\t=l = 0, 




{x,y) e 



(5) 



11 



where 



dxPdy^ 



(6) 



m I'f'rn. } 



{P:'?)ep 

V / dxPdyl ' 

{P:g)Gp 

rr (,A - - h 9irW,(v,u) _ av+iy 

"pq\") — dvpg — l^j=0^i dvpg ' ^P1~ dxPdyi- 
3.2.2. Gateaux Derivative of the Functional 

With the help of the adjoint equation, at each iteration the derivative of J with 
respect to aj{t) and bj{t) are as follows: 



dJ 



M 



da - -^j^j ~ f^Vmi'nWj{Um,Vm)dQ, j = 0,...,16, 



m=l 
M 

E 

m=l 



(7) 



§^ = fijbj - J2 In 0minvj(t;„, u^)dQ, j = 0, 16, 



where the adjoint functions (fm and (pm are the solutions to ([5]). 

3.2.3. Initialization 

Good initialization increases the approximation accuracy of the learnt PDEs. 
In our current implementation, we simply set the initial functions of u and v as 
the input image: 

Umix, y, 0) = Vm{x, y, 0) = Im{x, y), m = 1, 2, M. 

Then we employ a heuristic method to initialize the control functions. At each 
time step, is expected to be dmit) = '^"'"""^^^ so that Um{t) moves towards 
the expected output Om and by the form of (|Tj) we may solve {a^ (t)}]io ^^^^ that 

M „ 

V / [F{um,Vm, {a,{t)}]io) - dm{t)?dn (8) 
^-1 Jn 



m=l 
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is minimizecj^ In this way, we initialize aj{t) successively in time while fixing 
6,(t) = 0,j=0,l,...,16. 



3.2.4. Finite Difference Method for Numerical Solution 

To solve the intelligent PDE system numerically, we design a finite difference 

dt ■ 



scheme [[T8l for the PDEs. We discretize the PDEs, i.e. replace the derivatives — 



^ and 1^ with finite differences as follows: 

ax ox-^ 



^ df _ f(t+At)-f(t) 



dt At ' 

dl ^ f{x+l)-f{x) /QN 

dx 2 ' 

dx'^ 



f{x-l)-2f{x) + f{x + l) 



The discrete forms of 1^, and -^-4- can be defined similarly. In addition, we 

ay ' ay' axay •' 

discretize the integrations as 

where N is the number of pixels in the spatial area. At is a properly chosen time 
step size and T,„ = + 0.5J is the index of the expected output time. Then we 
use an explicit scheme to compute the numerical solutions. 

3.3. The Optimal-Control-Based Training Framework 

We now summarize in Algorithm [T] the data-based optimal control training 
framework for the intelligent PDE system. After the PDE system is learnt, it can 
be applied to new test images by solving ([T]), whose inputs and are both the 
test image and the solution u(t)\t=i is the desired output image. 

^It is to minimize the difference between the left and the right hand sides of |T|. 
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Algorithm 1 (Data-based optimal control framework for training the PDE 

system) 

Input: Training image pairs {(/„, O^)}^^^. 
1: Initialize aj(t), t = 0, At, 1 — At, by minimizing ^ and fix bj(t) = 0, 

J = 0,1,. ..,16. 
2: while not converged do 
3: Compute §^ and j = 0, 16, using 

4: Decide the search direction using the conjugate gradient method [fT9ll . 

5: Perform golden search along the search direction and update aj{t) and 

6,(t),j = 0,...,16. 
6: end while 

Output: The coefficient functions {aj(t)}]io ^^'^ {^ji^)}]=o- 

4. Experimental Results 

In this section, we apply our data-based optimal control framework to learn 
PDEs for four groups of basic computer vision problems: Natural image denois- 
ing, edge detection, blurring and deburring, and image segmentation and object 
detection. As our goal is to show that the data-based optimal control framework 
could be a new approach for designing PDEs and an effective regressor for many 
computer vision tasks, NOT to propose better algorithms for these tasks, we are 
not going to fine tune our PDEs and then compare it with the state-of-the-art al- 
gorithms in every task. 

4.1. Learning from Ground Truth: Natural Image Denoising 

Image denoising is one of the most fundamental low-level vision problems. 
For this task, we compare our learnt PDEs with the existing PDE-based denoising 
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methods, ROF [|5l and TV-L^ on images with unknown natural noise. This 
task is designed to demonstrate that our method can solve problems by learning 
from the ground truth. This is the first advantage of our data-based optimal control 
model. We take 240 images, each with a size of 150 x 150 pixels, of 11 objects 
using a Canon SOD digital camera, setting its ISO to 1600. For each object, 30 
images are taken without changing the camera settings (by fixing the focus, aper- 
ture and exposure time) and without moving the camera position. The average 
image of them can be regraded as the noiseless ground truth image. We randomly 
choose 8 objects. For each object we randomly choose 5 noisy images. These 
noisy images and their ground truth images are used to train the PDF system. 
Then we compare our learnt PDFs with the traditional PDFs in Q and TV-L^ ^ 
on images of the remaining 3 objects. 

Fig. [2] shows the comparison results. One can see that the PSNRs of our intelli- 
gent PDFs are dramatically higher than those of traditional PDFs. This is because 
our data-based PDF learning framework can easily adapt to unknown types of 
noise and obtain PDF forms to fit for the natural noise well, while most traditional 
PDF-based denoising methods were designed under specific assumptions on the 
types of noise (e.g., ROF is designed for Gaussian noise [5] while TV-L^ is de- 
signed for impulsive noise [|20l ). Therefore, they may not fit for unknown types 
of noise as well as our intelligent PDFs. The curves of the learnt coefficients for 
image denoising are shown in Fig. [3] 

4.2. Learning from Other Methods: Edge Detection 

The image edge detection task is used to demonstrate that our PDFs can be 
learnt from the results of different methods and achieve a better performance than 
all of them. This is another advantage of our data-based optimal control model. 
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27.D8dB 28.{i7dB 2r;.r)2dB ;12.2MB 
(a) (b) (c) (d) (e) 

Figure 2: The results of denoising images with natural noise, (a) original noiseless image, (b) 
noisy image with real noise, (c)-(e) denoised images using the ROF, TV-L^, and our intelligent 
PDE system, respectively. The PSNRs are presented below each image. 




i ° t i ° t 

(a) (b) 

Figure 3: Learnt coefficients of the intelligent PDE system for the natural noise denoising problem, 
(a) the coefficients {aj(t)}]lo '^^e image evolution, (b) the coefficients {&j(i)}]lo 
indicator evolution. 
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For this task, we use three simple first order edge detectors [211 (Sobel, Roberts 
Cross, and Prewitt) to generate the training data. We randomly choose 7 images 
from the Berkeley image database [|22l and use the above three detectors to gen- 
erate the output image^ together with the input images, to train our PDE system 
for edge detection. 

Fig. |4] shows part of the edge detection results on other images in the Berkeley 
image database. One can see that our PDEs respond selectively to edges and 
basically produce visually significant edges, while the edge maps of other three 
detectors are more chaotic. Note that the solution to our PDEs is supposed to 
be a more or less smooth function. So one cannot expect that our PDEs produce 
an exactly binary edge map. Instead, an approximation of a binary edge map is 
produced. The curves of the learnt coefficients for edge detection are shown in 
Fig.m 

4.3. Learning to Solve Both Primal and Inverse Problems: Blurring and Deblur- 
ring 

The traditional PDEs for solving different problems are usually of very dif- 
ferent appearance. The task of solving both blurring and deblurring is designed 
to show that the same form of PDEs can be learnt to solve both the primal and 
inverse problems. This is the third advantage of our data-based optimal control 
model. 

For the image blurring task (the primal problem), the output image is the con- 
volution of the input image with a Gaussian kernel. So we generate the output 

^This implies that we actually use a kind of combination of the results from different methods 
to train our PDE system. 
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i ° t i ° t 

(a) (b) 

Figure 5: Learnt coefficients of the intelligent PDE system for the edge detection problem, (a) the 
coefficients {aj{t)}^^Q for the image evolution, (b) the coefficients {6j(i)}j5,o fo'" '^^e indicator 
evolution. 
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images by blurring high resolution images using a Gaussian kernel with a = 1. 
The original images are used as the input. As shown in the third row of Fig. [6} 
the output is nearly identical to the ground truth (the second row of Fig. |6]). For 
the image deblurring task (the inverse problem), we just exchange the input and 
output images for training. One can see in the bottom row of Fig.[6]that the output 
is very close to the original image (first row of Fig. |6]). The curves of the learnt 
coefficients for image deblurring are shown in Fig.|7] 




23.iidB 21.23dB 2'>.u:)dB 2^.-'',2dB 2Lr>;5dB 




2-2.7SJdB 23.7T6ti 25.23dB 25.72dB -_>[.-_'-AlB 























^K. fit'' 










27.£)()dB 


28.47dB 


29.94dB 


29.J7dB 


25.50dB 



Figure 6: The results of image blurring and deblurring. The top row are the original images. The 
second row are the blurring results of a Gaussian kernel. The third row are the blurring results of 
our Intelligent PDE. The bottom row are the deblurring results of our Intelligent PDE. The PSNRs 
are presented below each image. 
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Figure 7: Learnt coefficients of the intelligent PDE system for image blurring (top row) and deblur- 
ring (bottom row) problems, (a) and (c) are the coefficients {aj(i)}j^Q for the image evolutions, 
(b) and (d) are the coefficients {&j (i)}]=o indicator evolutions. 
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4.4. Learning from Humans: Image Segmentation and Object Detection 

Image segmentation and object detection are designed to demonstrate that our 
PDE system can learn from the human behavior directly (learn the segmentation 
and detection results provided by humans, e.g., manually segmented masks). 

For image segmentation, it is a highly ill-posed problem and there are many 
criteria that define the goal of segmentation, e.g., breaking an image into regions 
with similar intensity, color, texture, or expected shape. As none of the current 
image segmentation algorithms can perform object level segmentation well out 
of complex backgrounds, we choose to require our PDEs to achieve a reasonable 
goal, namely segmenting relatively darker objects against relatively simple back- 
grounds, where both the foreground and the background can be highly textured 
and simple thresholding cannot separate them. So we select 60 images from the 
Corel image database ||23l that have relatively darker foregrounds and relatively 
simple backgrounds, but the foreground is not of uniformly lower graylevels than 
the background, and also prepare the manually segmented binary masks as the out- 
puts of the training images, where the black regions are the backgrounds (Fig. [8]). 




Figure 8: Examples of the training images for image segmentation. In each group of images, on 
the left is the input image and on the right is the ground truth output mask. 

Part of the segmentation results are shown in Fig. |9| where we have set a 
threshold for the output mask maps of our learnt PDEs with a constant 0.5. We 
see that our learnt PDEs produce fairly good object masks. We also test the active 
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contour method by Li et al. flO^and the normahzed cut method [|24]j ^ One can 
see from Fig. |9] that the active contour method cannot segment object details due 
to the smoothness constraint on the object shape and the normalized cut method 
cannot produce a closed foreground region. To provide a quantitative evaluation, 
we use the F-measures that merge the precision and recall of segmentation: 

(1 + a) ■ recall ■ precision 

Fa = —. , 7-1 — ' where 

a ■ precision + recall 

„ Un5| . . \AnB\ 

recall = — — - — , precision 



\A\ ' \B\ ' 

in which A is the ground truth mask and B is the computed mask. The most 
common choice of a is 2. On our test images, the F2 measures of our PDEs, [fTOll 
and ial are 0.90 ± 0.05, 0.83 ± 0.16 and 0.61 ± 0.20, respectively. One can see 
that the performance of our PDEs is better than theirs, in both visual quality and 
quantitative measure. The curves of the learnt coefficients for image segmentation 



are shown in Fig. 10 



We also present the evolution process of the mask maps across time (Fig. 1 1 1. 
One can see that although the foreground is relatively darker than the background, 
the PDEs correctly detect the most salient points/edges and then propagate the 
information across the foreground region, resulting in a brighter output region for 
the foreground. 

Now we apply our intelligent PDEs for a more complex task: Object detection. 
Namely, the PDEs should respond strongly to the object of interest while not 
responding (or responding much more weakly) if the object is absent in the image. 
It should be challenging enough for one to manually design PDEs to perform 



'*Code available at 
^Code available at 



http://www.engr.uconn.edu/~cmli/ 



http://www.cis.upenn.edu/~jshi/software/ 
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Figure 9: The results of image segmentation. The top row are the segmentation results of active 
contour ifTOl on the original images. The bottom row are the results obtained by our IntelUgent 
PDE. 
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(a) (b) 

Figure 10: Learnt coefficients of the intelligent PDE system for image segmentation problems, 
(a) the coefficients {aj(t)}]lo '^^e image evolution, (b) the coefficients {&j(i)}]lo 
indicator evolution. 
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Figure 11: The evolution of the mask maps. For each row, the first image is the input image, the 
second to the fifth are the mask maps at time t = 0.25, 0.50, 0.75, 1.0, respectively, and the last 
image is the final mask map with a threshold of 0.5. 



such a problem. As a result, we are unaware of any PDE-based method that can 
accomplish this task. The existing PDE-based segmentation algorithms always 
output an "object region" even if the image does not contain the object of interest. 
In contrast, we will show that as desired our PDEs are able to respond selectively. 
We choose the "plane" data set in Corel ||23l. We select 30 images from this data 
set as positive samples and also prepare 30 images without the object of interest 
as negative samples. We also provide their ground truth object mask^in order to 
complete the training data. 



In Fig. 1 2 one can see that our learnt PDEs respond well to the objects of inter- 
est (first three images), while the response to images without the objects of interest 
is relatively low across the whole images (last two images). It seems that our PDEs 
automatically identify that the concurrent high-contrast edges/junctions/comers 
are the key features of planes. The above examples show that our learnt PDEs 
are able to differentiate the object/non-object regions, without requiring the user 



*For the positive samples, we manually segment binary masks as the output images. For the 
negative samples, the ground truth output masks are all-zero images. 
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to teach them what features are and what factors to consider. The curves of the 
learnt coefficients for object detection are shown in Fig. [13 




Figure 12: The results of "plane" detection. The top row are the original images, the first three are 
images containing planes and the last two are without planes. The bottom row are the detection 
results of our intelligent PDE for images with and without planes, respectively. 
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Figure 13: Learnt coefficients of the intelligent PDE system for the plane detection problem, 
(a) the coefficients {aj(t)}]lo ^'^^ '■^^ image evolution, (b) the coefficients {&j(i)}]lo f^'" '■^^ 
indicator evolution. 

5. Conclusion 

In this paper, we have presented a framework for using data-based optimal 
control to learn PDEs as a general regressor to approximate the nonlinear map- 
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pings of different visual processing tasks. The experimental results on some com- 
puter vision and image processing problems show that our framework is promis- 
ing. However, the current work is still preliminary, so we plan to improve and 
enrich our work in the following aspects. First, more theoretical issues should be 
addressed for this PDE system. For example, we will try to apply the Adomian 
decomposition method [|25l to express the exact analytical solution to ([T]) and then 
analyze its physical properties. Second, we would like to develop more com- 
putationally efficient numerical algorithms to solve our PDF-constrained optimal 
control problem (|4]). Third, we will apply our framework to more vision tasks to 
find out to what extent it works. 



Appendix A. Proof of Property 2.1 



We prove that the coefficients {aj}]2=o and {6j}]^Q must be independent of 

{x,y). 



Proof. We prove for F{u,v, {aj}]=o) in ([T| only. We may rewrite 

F{u, V, {aj}]%) = F{u, V, X, y, t). 

Then it suffices to prove that F is independent of (x, y). 

By the definition of translational invariance, when I{x, y) changes to I{x — 
Xo, y—yo) by shifting with a displacement (xq, yo), u{x, y) and f (x, y) will change 
to m(x — xo, y — yo) andf (x — xo, y — yo), respectively. So thepair m(x — xq, y — yo) 
and v{x — xo,y — yo) fulfils ([T]), i.e., 

du{x -xo,y- yo) 
dt 

-F{u{x -xo,y- yo),v{x -xo,y- yo),x,y,t) = 0. 
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Next, we replace (x — Xo^y — i/q) in the above equation with (x, y) and have: 

— ^ F{u{x, y),v{x, y),x + xo,y + yo, t) = 0. 

On the other hand, the pair y), v{x, y)) also fulfils ([T]), i.e., 

du{x, y) 



dt 



F{u{x,y),v{x,y),x,y,t) = 0. 



Therefore, F{u, v^x + xo,y + yo, t) = F{u, f , x, y, t), \/{xo, yo) that confines the 
input image inside Vl. So F is independent of {x, y). □ 

Acknowledgment 

This work was partially supported by the grants of the National Science Foun- 
dation of China, No. U0935004 and 60873181. 

References 

[1] T. Chen and J. Shen, Image processing and analysis: variational, PDE, 
wavelet, and stochastic methods. SIAM Publisher, 2005. 

[2] P. Pietro and M. Jitendra, "Scale-space and edge detection using anisotropic 
diffusion," IEEE Transactions on Pattern Analysis and Machine Intelligence, 
vol. 12, pp. 629-639, 1990. 

[3] S. Osher and L. Rudin, "Feature-oriented image enhancement using shock 
filters," SIAM Journal on Numerical Analysis, vol. 27, pp. 919-940, 1990. 

[4] G. Sapiro, Geometric partial differential equations and image analysis. 
Cambridge University Press, 2001. 



27 



[5] L. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise 
removal algorithms," Physica D., vol. 60, pp. 259-268, 1992. 

[6] T. Chan and S. Esedoglu, "Aspects of total variation regularizaed function 
approximation," SIAM Journal on Applied Mathematics, vol. 65, pp. 1817- 
1837, 2005. 

[7] D. Strong and T. Chan, "Edge-preserving and scale-dependent properties of 
total variation regularization," Inverse Problems, vol. 19, pp. 165-187, 2003. 

[8] X. Bresson, S. Esedoglu, P. Vandergheynst, J. -P. Thiran, and S. Osher, "Fast 
gobal minimization of the active contour/snake model," Journal of Mathe- 
matical Imaging and Vision, vol. 28. 

[9] A. ChamboUe, "Total variation minimization and a class of binary MRF 

models," in Energy Minimization Methods in Computer Vision and Pattern 
Recognition (EMMCVPR), 2008. 

[10] C. Li, C. Xu, C. Gui, and M. Fox, "Level set evolution without re- 
initialization: a new variational formulation," in IEEE Conference on Com- 
puter Vision and Pattern Recognition ( CVPR), 2005. 

[11] X. Gao, B. Wang, D. Tao, and X. Li, "A relay level set method for automatic 
image segmentation," IEEE Transactions on Systems, Man and Cybernetics 
PartB: Cybernetics, vol. 41, pp. 518-525, 2011. 

[12] D. Cheng, Field and wave electromagnetics (2nd Edition). Prentics-Hall, 
1989. 



28 



[13] P. Olver, Applications of Lie groups to differential equations. Springer- 
Verlarg, 1993. 

[14] D. Kirk, Optimal control theory: an introduction. Prentice-Hall, 1971. 

[15] A. Ababnah and B. Natarajan, "Optimal contiol-based stiategy for sensor 
deployment," IEEE Transactions on Systems, Man and Cybernetics Part B: 
Cybernetics, vol. 41, pp. 97-104, 2011. 

[16] J. Lions, Optimal control systems governed by partial differential equations. 
Springer- Verlag, 1971. 

[17] R. Liu, Z. Lin, W. Zhang, and Z. Su, "Learning PDEs for image restoration 
via optimal control," in European Conference on Computer Vision (ECCV), 
2010. 

[18] A. Jain, "Partial differential equations and finite-difference methods in im- 
age processing, part 1," Journal of Optimization Theory and Applications, 
vol. 23, pp. 65-91, 1977. 

[19] J. Stoer and R. Bulirsch, Introduction to numerical analysis (2nd Edition). 
Springer- Verlarg, 1998. 

[20] N. Mila, "A variational approach to reomve outliers and impulse noise," 
Journal of Mathematical Imaging and Vision, vol. 20, pp. 99-120, 2004. 

[21] J. Parker, Algorithms for image processing and computer vision. John Wiley 
& Sons Inc, 1997. 



29 



[22] D. R. Martin, C. Fowlkes, D. Tal, and J. Malik, "A database of human seg- 
mented natural images and its application to evaluating segmentation algo- 
rithms and measuring ecological statistics," in International Conference on 
Computer Vision (ICCV), 2001. 

[23] Corel photo library, corel corp., Ottawa, Canada. 

[24] J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans- 
actions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 
888-905, 2000. 

[25] A. Wazwaz, Partial differential equations and solitary waves theory. 
Springer- Verlag, 2009. 



30 



